Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vendor go dependencies #2403

Merged
merged 10 commits into from
Jun 25, 2024
Merged

Vendor go dependencies #2403

merged 10 commits into from
Jun 25, 2024

Conversation

pajakd
Copy link
Contributor

@pajakd pajakd commented Jun 12, 2024

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

I ran go mod vendor in the main directory. I would like to verify if having the dependencies vendored shortens build time (to help decide in the discussion #2374).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 12, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @pajakd. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 12, 2024
@netlify Netlify
Copy link

netlify bot commented Jun 12, 2024 •
edited
Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit 125d92f
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6672b091cd8f4400089d80fa

@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 12, 2024
@tenzen-y
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 12, 2024
@alculquicondor
Copy link
Contributor

It looks like commands like this:

https://github.com/kubernetes-sigs/kueue/blob/6ecaa877ee65791e6333f147fafcbb7888039a03/Makefile-deps.mk#L99C20-L99C84

Are not pointing to the vendor folder

@pajakd
Copy link
Contributor Author

pajakd commented Jun 13, 2024

Yes, this command returns an empty string if the vendor directory is present.

What is more problematic is that /vendor directory only contains Go-packages (golang/go#26366). So, the files (the CRD .yaml definitions) that we are trying to copy here are not in the vendored package directory.

@pajakd
Copy link
Contributor Author

pajakd commented Jun 14, 2024

/retest

@pajakd
Copy link
Contributor Author

pajakd commented Jun 14, 2024

/retest

@pajakd
Copy link
Contributor Author

pajakd commented Jun 14, 2024

So, it looks like it decreases the runtime of build-image-main target (hard to assess exactly as the runtimes vary quite significantly from run to run).

Command make verify, updated a number of files in /vendor directory -- I committed the changes and only then verify-main passed. However, if I try to run go mod vendor now it would revert the changes made by make verify. Not sure what to do with it.

@alculquicondor
Copy link
Contributor

Re: make verify

We probably need to change the verify rules to exclude the vendor folder. It might be formatting or doing similar things.

alculquicondor
alculquicondor reviewed Jun 18, 2024
View reviewed changes
Dockerfile Outdated
Comment on lines 10 to 12
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove these 2 lines too

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@alculquicondor
Copy link
Contributor

Also need to update https://github.com/kubernetes-sigs/kueue/blob/main/cmd/importer/Dockerfile

@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024

/retest

@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024

/retest

@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024

/retest

@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024 •
edited
Loading

"There are no nodes that your pod can schedule" -- looks like too many build jobs ran in parallel. Let me retry.

@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024

/retest

@pajakd pajakd marked this pull request as ready for review June 19, 2024 10:21
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 19, 2024
@pajakd
Copy link
Contributor Author

pajakd commented Jun 19, 2024

Fixed the problems with verify (had to exclude /vendor directory from gofmt).

It looks like the speedup in build time is small (or none).

The only benefit from vendored dependencies would be more hermetic build.

@alculquicondor
Copy link
Contributor

/test pull-kueue-build-image-main

(curious if it's consistently faster)

@alculquicondor
Copy link
Contributor

/test pull-kueue-build-image-main

@alculquicondor
Copy link
Contributor

It looks like the builds are between 20 to 30s faster.

@tenzen-y wdyt?

@alculquicondor
Copy link
Contributor

I think we also need a change in .github/dependabot.yaml https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#vendor

@pajakd
Copy link
Contributor Author

pajakd commented Jun 20, 2024 •
edited
Loading

As far as I understand with our configuration (package-ecosystem: "gomod") we don't need to change anything. From the documentation "Don't use this option if you're using gomod as Dependabot automatically detects vendoring for this tool.". Additional reference: https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/

@alculquicondor alculquicondor mentioned this pull request Jun 20, 2024
Closed
3 tasks
@alculquicondor
Copy link
Contributor

/lgtm
/approve

@alculquicondor
Copy link
Contributor

/hold cancel

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 25, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 7ac2f43ec02fa39c02831222f9456b4929058c11

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, pajakd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 25, 2024
@tenzen-y
Copy link
Member

It looks like the builds are between 20 to 30s faster.

@tenzen-y wdyt?

I understand the valuables of the vendor.

@alculquicondor
Copy link
Contributor

/retest
Flaky #2429

@k8s-ci-robot k8s-ci-robot merged commit bb6ca65 into kubernetes-sigs:main Jun 25, 2024
16 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v0.8 milestone Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alculquicondor alculquicondor alculquicondor left review comments

@mimowo mimowo Awaiting requested review from mimowo

@denkensk denkensk Awaiting requested review from denkensk

@kerthcet kerthcet Awaiting requested review from kerthcet

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
v0.8
Development

Successfully merging this pull request may close these issues.
4 participants
@pajakd @k8s-ci-robot @tenzen-y @alculquicondor